feat: ResultSet& ValueWrapper sugars for Graph Visualisation & Data Sci #323

wey-gu · 2024-03-15T14:22:07Z

implement: #318, #324

What type of PR is this?

bug
feature
enhancement

What problem(s) does this PR solve?

Issue(s) number: #318 , #324

Description:

Docs polished, straightforwardly getting started
sugars to better handle ResultSet for Graph Vis/Rendering && as Pandas Dataframe
Optimized Config, now user could omit them to enable ease of first-time try, fixed SessionPool Config Check bug

`resultSet.dict_for_vis()`

easily get node list/ edge list and more information in ANY query as a result.

result = session.execute('GET SUBGRAPH WITH PROP 2 STEPS FROM "player101" YIELD VERTICES AS nodes, EDGES AS relationships;')
v = result.dict_for_vis()

# v is like:
{
    'nodes': [
        {
            'id': 'player100',
            'labels': ['player'],
            'props': {
                'name': 'Tim Duncan', 
                'age': '42', 
                'id': 'player100'
            }
        },
        {
            'id': 'player101',
            'labels': ['player'],
            'props': {
                'age': '36', 
                'name': 'Tony Parker', 
                'id': 'player101'
            }
        }
    ],
    'edges': [
        {
            'src': 'player100',
            'dst': 'player101',
            'name': 'follow',
            'props': {
                'degree': '95'
            }
        }
    ],
    'nodes_dict': {
        'player100': {
            'id': 'player100',
            'labels': ['player'],
            'props': {
                'name': 'Tim Duncan', 
                'age': '42', 
                'id': 'player100'
            }
        },
        'player101': {
            'id': 'player101',
            'labels': ['player'],
            'props': {
                'age': '36', 
                'name': 'Tony Parker', 
                'id': 'player101'
            }
        }
    },
    'edges_dict': {
        ('player100', 'player101', 0, 'follow'): {
            'src': 'player100',
            'dst': 'player101',
            'name': 'follow',
            'props': {
                'degree': '95'
            }
        }
    },
    'nodes_count': 2,
    'edges_count': 1
}

`resultSet.as_pandas()`

Get pandas dataframe with all data primitive typed

In [3]: result = session.execute('GET SUBGRAPH WITH PROP 2 STEPS FROM "player101" YIELD VERTICES AS nodes, EDGES AS relationships
   ...: ;')

In [4]: df = result.as_pandas()

In [5]: df
Out[5]: 
                                               nodes                                      relationships
0  [{'vid': 'player101', 'tags': ['player'], 'pro...  [{'src': 'player101', 'dst': 'team204', 'type'...
1  [{'vid': 'team215', 'tags': ['team'], 'props':...  [{'src': 'player111', 'dst': 'team215', 'type'...
2  [{'vid': 'player146', 'tags': ['player'], 'pro...  [{'src': 'player146', 'dst': 'team222', 'type'...

In [7]: x = df.iloc[0, 0]

In [8]: type(x)
Out[8]: list

In [9]: x
Out[9]: 
[{'vid': 'player101',
  'tags': ['player'],
  'props': {'age': 36, 'name': "Tony Parker"}}]

In [10]: type(x[0])
Out[10]: dict

result = session.execute('GET SUBGRAPH WITH PROP 2 STEPS FROM "player101" YIELD VERTICES AS nodes, EDGES AS relationships;') v = result.dict_for_vis() implement: #318

wey-gu · 2024-03-15T14:31:12Z

TODOS:

docs/ example, via c303cad
test, via c303cad
as pandas dataframe, where all cells are primarily typed(no nebula bytes) , via 74469cd
Example of Frontend, via 8ce2af0

wey-gu · 2024-03-16T03:15:42Z

example/FormatResp.py

@@ -29,7 +42,7 @@ def result_to_df(result: ResultSet) -> pd.DataFrame:
        col_name = columns[col_num]
        col_list = result.column_values(col_name)
        d[col_name] = [x.cast() for x in col_list]
-    return pd.DataFrame.from_dict(d, columns=columns)


there were errors here.

wey-gu · 2024-03-16T03:16:01Z

example/GraphClientSimpleExample.py

@@ -59,6 +60,28 @@
        assert resp.is_succeeded(), resp.error_msg()
        print_resp(resp)

+        # query data


added coverage of those examples

codecov-commenter · 2024-03-16T03:29:20Z

Codecov Report

Attention: Patch coverage is 44.34783% with 64 lines in your changes are missing coverage. Please review.

Project coverage is 75.61%. Comparing base (effa675) to head (fc65c90).

Files	Patch %	Lines
nebula3/data/ResultSet.py	4.83%	59 Missing ⚠️
nebula3/data/DataObject.py	90.90%	4 Missing ⚠️
nebula3/gclient/net/ConnectionPool.py	80.00%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #323      +/-   ##
==========================================
- Coverage   76.90%   75.61%   -1.29%     
==========================================
  Files          18       18              
  Lines        2516     2625     +109     
==========================================
+ Hits         1935     1985      +50     
- Misses        581      640      +59

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

wey-gu · 2024-03-16T03:46:53Z

@Nicole00 @haoxins what do you think, please?

in a straightforward way

- Readme getting started polished - Config should be able to omit - The previous SessionPoolConfig Check mechnism was wrong, fixed it

Nicole00 · 2024-03-18T02:14:22Z

nebula3/data/ResultSet.py

@@ -192,6 +192,187 @@ def rows(self):
            return []
        return self._data_set_wrapper.get_rows()

+    def dict_for_vis(self):


what about the result value is just a path?

Yes, it hands Path use cases, too(even a list of paths ideally), it literally hands all results that are feasible to be rendered.

For Paths, it'll extract edge list and node list, too. For all graph Vis Lib, edge list and node list are all they needed.

haoxins · 2024-03-18T03:42:18Z

maybe we can move example/apache_echarts.html into a separate repo?
After all, this is the client for Python.
And we can make it as a SPA by React/Vue/Svelte/Next.js and even more.

nebula3/data/ResultSet.py

wey-gu · 2024-03-18T05:52:18Z

maybe we can move example/apache_echarts.html into a separate repo? After all, this is the client for Python. And we can make it as a SPA by React/Vue/Svelte/Next.js and even more.

I think we could for now put it here as the self-contained hint, or in a gist to refer it to, but YES, we could further create FE libs to work with it! What do you think?

haoxins · 2024-03-18T06:15:37Z

maybe we can move example/apache_echarts.html into a separate repo? After all, this is the client for Python. And we can make it as a SPA by React/Vue/Svelte/Next.js and even more.

I think we could for now put it here as the self-contained hint, or in a gist to refer it to, but YES, we could further create FE libs to work with it! What do you think?

👍

HarrisChu · 2024-03-18T06:48:29Z

how about we do this in a new repo, e.g. graph-sci ? and it could generate dafaframe from:

nebula-graph
neo4j
ISO-GQL in future?

#example 
sci = load_wrapper("nebula")
df = sci.to_df(result)
xxx

wey-gu · 2024-03-18T10:21:52Z

how about we do this in a new repo, e.g. graph-sci ? and it could generate dafaframe from:

nebula-graph

neo4j

ISO-GQL in future?
#example 

sci = load_wrapper("nebula")

df = sci.to_df(result)

xxx

Yes we can!

And that one would be super function complete and with proper dependencies with those sci toolchain.

In this repo I kept the minimal feature workable with opinionated defaults(not tweak-able), yet with all dependencies import in function, thus our client won't depend on pandas etc...

haoxins · 2024-03-18T15:54:38Z

And that one would be super function complete and with proper dependencies with those sci toolchain.

Maybe it's not suitable to post in this thread, but I think it's worth to share the emerging project.

https://github.com/Pometry/Raphtory

Which has some great ideas!

wey-gu · 2024-03-18T18:10:47Z

And that one would be super function complete and with proper dependencies with those sci toolchain.

Maybe it's not suitable to post in this thread, but I think it's worth to share the emerging project.

https://github.com/Pometry/Raphtory

Which has some great ideas!

Lovely embedded one!quite inspiring for the API!

haoxins · 2024-03-25T02:52:38Z

example/apache_echarts.html

+  return {name: category};
+});
+
+option = {


It seems that the G6's APIs are more suitable for Graph special visualization?

https://github.com/antvis/G6

Make sense, I can provide mainstream libs flavor of output, including g6.

nebula3/data/ResultSet.py

feat: resultSet sugar for Graph Visulization

1970cd3

result = session.execute('GET SUBGRAPH WITH PROP 2 STEPS FROM "player101" YIELD VERTICES AS nodes, EDGES AS relationships;') v = result.dict_for_vis() implement: #318

pop rank to prop

baeb095

wey-gu force-pushed the result_for_vis branch from e4dc6b6 to baeb095 Compare March 15, 2024 14:33

feat: resultSet.as_pandas() && ValueWrapper.cast_primitive()

74469cd

wey-gu changed the title ~~feat: resultSet sugar for Graph Visulization~~ feat: ResultSet& ValueWrapper sugars for Graph Visualisation & Data Sci Mar 15, 2024

wey-gu added 6 commits March 15, 2024 17:59

docs and tests

c303cad

chore: polishing wording of contribution

881e9ae

docs: polishing readme

d3f83ea

fix: test cases missing import(sorry)

c4d89a3

fix: tag props calling failure and tests

d4acd12

fix test path primitive

9c1f685

wey-gu force-pushed the result_for_vis branch from 06ff0dc to db6b1d3 Compare March 16, 2024 03:12

wey-gu commented Mar 16, 2024

View reviewed changes

fix tests, add example tests

6413985

wey-gu force-pushed the result_for_vis branch from db6b1d3 to 6413985 Compare March 16, 2024 03:18

fix time.cast_primitive()

3856af5

docs: polishing Getting Started

54f0237

in a straightforward way

wey-gu mentioned this pull request Mar 16, 2024

topic: Dev User Experience Revisit #324

Open

17 tasks

feat: Config and SessionPoolConfig could be omit by default

9f6bc54

- Readme getting started polished - Config should be able to omit - The previous SessionPoolConfig Check mechnism was wrong, fixed it

wey-gu requested review from Nicole00 and HarrisChu March 16, 2024 04:45

wey-gu added the doc affected PR: improvements or additions to documentation label Mar 16, 2024

wey-gu added 2 commits March 16, 2024 05:43

docs: graph vis with dict_for_vis & Apache ECharts

8ce2af0

fix: not run GraphVis example in CI

e15f217

Nicole00 reviewed Mar 18, 2024

View reviewed changes

haoxins reviewed Mar 18, 2024

View reviewed changes

nebula3/data/ResultSet.py Outdated Show resolved Hide resolved

address Hao Xin's comment

bf072ce

Merge branch 'master' into result_for_vis

54faf1c

wey-gu requested review from Nicole00 and haoxins March 22, 2024 10:02

haoxins reviewed Mar 25, 2024

View reviewed changes

Nicole00 reviewed Mar 25, 2024

View reviewed changes

nebula3/data/ResultSet.py Show resolved Hide resolved

Nicole00 approved these changes Apr 7, 2024

View reviewed changes

Merge branch 'master' into result_for_vis

fc65c90

haoxins approved these changes Apr 7, 2024

View reviewed changes

Merge branch 'master' into result_for_vis

5b71234

wey-gu mentioned this pull request Apr 22, 2024

api: introduce interface/function to better prepare data for graph vis #318

Open

2 tasks

Nicole00 merged commit 73f7697 into master Apr 22, 2024
10 checks passed

Nicole00 deleted the result_for_vis branch April 22, 2024 08:39

wey-gu mentioned this pull request Apr 27, 2024

Weekly Report 2024-04-26 vesoft-inc/nebula-community#434

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: ResultSet& ValueWrapper sugars for Graph Visualisation & Data Sci #323

feat: ResultSet& ValueWrapper sugars for Graph Visualisation & Data Sci #323

wey-gu commented Mar 15, 2024 •

edited

Loading

wey-gu commented Mar 15, 2024 •

edited

Loading

wey-gu Mar 16, 2024

wey-gu Mar 16, 2024

codecov-commenter commented Mar 16, 2024 •

edited

Loading

wey-gu commented Mar 16, 2024

Nicole00 Mar 18, 2024

wey-gu Mar 18, 2024

wey-gu Mar 18, 2024

haoxins commented Mar 18, 2024 •

edited

Loading

wey-gu commented Mar 18, 2024

haoxins commented Mar 18, 2024

HarrisChu commented Mar 18, 2024 •

edited

Loading

wey-gu commented Mar 18, 2024

haoxins commented Mar 18, 2024

wey-gu commented Mar 18, 2024

haoxins Mar 25, 2024

wey-gu Apr 5, 2024

feat: ResultSet& ValueWrapper sugars for Graph Visualisation & Data Sci #323

feat: ResultSet& ValueWrapper sugars for Graph Visualisation & Data Sci #323

Conversation

wey-gu commented Mar 15, 2024 • edited Loading

What type of PR is this?

What problem(s) does this PR solve?

Issue(s) number: #318 , #324

Description:

resultSet.dict_for_vis()

resultSet.as_pandas()

wey-gu commented Mar 15, 2024 • edited Loading

wey-gu Mar 16, 2024

Choose a reason for hiding this comment

wey-gu Mar 16, 2024

Choose a reason for hiding this comment

codecov-commenter commented Mar 16, 2024 • edited Loading

Codecov Report

wey-gu commented Mar 16, 2024

Nicole00 Mar 18, 2024

Choose a reason for hiding this comment

wey-gu Mar 18, 2024

Choose a reason for hiding this comment

wey-gu Mar 18, 2024

Choose a reason for hiding this comment

haoxins commented Mar 18, 2024 • edited Loading

wey-gu commented Mar 18, 2024

haoxins commented Mar 18, 2024

HarrisChu commented Mar 18, 2024 • edited Loading

wey-gu commented Mar 18, 2024

haoxins commented Mar 18, 2024

wey-gu commented Mar 18, 2024

haoxins Mar 25, 2024

Choose a reason for hiding this comment

wey-gu Apr 5, 2024

Choose a reason for hiding this comment

wey-gu commented Mar 15, 2024 •

edited

Loading

`resultSet.dict_for_vis()`

`resultSet.as_pandas()`

wey-gu commented Mar 15, 2024 •

edited

Loading

codecov-commenter commented Mar 16, 2024 •

edited

Loading

haoxins commented Mar 18, 2024 •

edited

Loading

HarrisChu commented Mar 18, 2024 •

edited

Loading